{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "K5de6EUfWWUb"
   },
   "source": [
    "# TensorFlow2.0教程-自动求导\n",
    "\n",
    "这节我们会介绍使用tensorflow2自动求导的方法。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 35
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 2148,
     "status": "ok",
     "timestamp": 1559219648589,
     "user": {
      "displayName": "Will Chen",
      "photoUrl": "",
      "userId": "01179718990779759737"
     },
     "user_tz": -480
    },
    "id": "49BpOoOXWa8C",
    "outputId": "6aa76be9-e27c-4b1c-e751-7217f394efbc"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2.0.0-alpha0\n"
     ]
    }
   ],
   "source": [
    "from __future__ import absolute_import, division, print_function, unicode_literals\n",
    "# !pip uninstall tensorflow\n",
    "#!pip install tensorflow==2.0.0-alpha\n",
    "import tensorflow as tf\n",
    "print(tf.__version__)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "qpi0MAurTzfy"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "YqfVrbeTQupu"
   },
   "source": [
    "## 一、Gradient tapes\n",
    "tensorflow 提供tf.GradientTape api来实现自动求导功能。只要在tf.GradientTape()上下文中执行的操作，都会被记录与“tape”中，然后tensorflow使用反向自动微分来计算相关操作的梯度。\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 71
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 760,
     "status": "ok",
     "timestamp": 1559219653881,
     "user": {
      "displayName": "Will Chen",
      "photoUrl": "",
      "userId": "01179718990779759737"
     },
     "user_tz": -480
    },
    "id": "S2dV3uiKQZsE",
    "outputId": "1dcd5f41-ee73-4eed-837d-b45638fc63e3"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tf.Tensor(\n",
      "[[8. 8.]\n",
      " [8. 8.]], shape=(2, 2), dtype=float32)\n"
     ]
    }
   ],
   "source": [
    "x = tf.ones((2,2))\n",
    "\n",
    "# 需要计算梯度的操作\n",
    "with tf.GradientTape() as t:\n",
    "    t.watch(x)\n",
    "    y = tf.reduce_sum(x)\n",
    "    z = tf.multiply(y,y)\n",
    "# 计算z关于x的梯度\n",
    "dz_dx = t.gradient(z, x)\n",
    "print(dz_dx)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "fPn28EA3Uqp6"
   },
   "source": [
    "也可以输出对中间变量的导数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 35
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 942,
     "status": "ok",
     "timestamp": 1559219655992,
     "user": {
      "displayName": "Will Chen",
      "photoUrl": "",
      "userId": "01179718990779759737"
     },
     "user_tz": -480
    },
    "id": "C-D2Lf06TLgc",
    "outputId": "6b6e57d1-3540-4839-d877-0818733da407"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tf.Tensor(8.0, shape=(), dtype=float32)\n"
     ]
    }
   ],
   "source": [
    "# 梯度求导只能每个tape一次\n",
    "with tf.GradientTape() as t:\n",
    "    t.watch(x)\n",
    "    y = tf.reduce_sum(x)\n",
    "    z = tf.multiply(y,y)\n",
    "    \n",
    "dz_dy = t.gradient(z, y)\n",
    "print(dz_dy)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "5Ntrsuv1tewy"
   },
   "source": [
    "默认情况下GradientTape的资源会在执行tf.GradientTape()后被释放。如果想多次计算梯度，需要创建一个持久的GradientTape。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 89
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 766,
     "status": "ok",
     "timestamp": 1559219848611,
     "user": {
      "displayName": "Will Chen",
      "photoUrl": "",
      "userId": "01179718990779759737"
     },
     "user_tz": -480
    },
    "id": "5kZoUqbrVAO5",
    "outputId": "65806f79-3d13-492f-fa1b-29e9c86a1566"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tf.Tensor(\n",
      "[[8. 8.]\n",
      " [8. 8.]], shape=(2, 2), dtype=float32)\n",
      "tf.Tensor(8.0, shape=(), dtype=float32)\n"
     ]
    }
   ],
   "source": [
    "with tf.GradientTape(persistent=True) as t:\n",
    "    t.watch(x)\n",
    "    y = tf.reduce_sum(x)\n",
    "    z = tf.multiply(y, y)\n",
    "    \n",
    "dz_dx = t.gradient(z,x)\n",
    "print(dz_dx)\n",
    "dz_dy = t.gradient(z, y)\n",
    "print(dz_dy)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "meFCHe37wBc8"
   },
   "source": [
    "## 二、记录控制流\n",
    "因为tapes记录了整个操作，所以即使过程中存在python控制流（如if， while），梯度求导也能正常处理。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 71
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 1517,
     "status": "ok",
     "timestamp": 1559220443136,
     "user": {
      "displayName": "Will Chen",
      "photoUrl": "",
      "userId": "01179718990779759737"
     },
     "user_tz": -480
    },
    "id": "OP5XoSJovsJs",
    "outputId": "b20ae483-321c-4a15-9e76-7a4773137ed3"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tf.Tensor(12.0, shape=(), dtype=float32)\n",
      "tf.Tensor(12.0, shape=(), dtype=float32)\n",
      "tf.Tensor(4.0, shape=(), dtype=float32)\n"
     ]
    }
   ],
   "source": [
    "def f(x, y):\n",
    "    output = 1.0\n",
    "    # 根据y的循环\n",
    "    for i in range(y):\n",
    "        # 根据每一项进行判断\n",
    "        if i> 1 and i<5:\n",
    "            output = tf.multiply(output, x)\n",
    "    return output\n",
    "\n",
    "def grad(x, y):\n",
    "    with tf.GradientTape() as t:\n",
    "        t.watch(x)\n",
    "        out = f(x, y)\n",
    "        # 返回梯度\n",
    "        return t.gradient(out, x)\n",
    "# x为固定值\n",
    "x = tf.convert_to_tensor(2.0)\n",
    "\n",
    "print(grad(x, 6))\n",
    "print(grad(x, 5))\n",
    "print(grad(x, 4))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "EihwzFwIyXKX"
   },
   "source": [
    "## 三、高阶梯度\n",
    "GradientTape上下文管理器在计算梯度的同时也会保持梯度，所以GradientTape也可以实现高阶梯度计算，"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 53
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 1144,
     "status": "ok",
     "timestamp": 1559221001653,
     "user": {
      "displayName": "Will Chen",
      "photoUrl": "",
      "userId": "01179718990779759737"
     },
     "user_tz": -480
    },
    "id": "CCgxtUbNx5s_",
    "outputId": "3b68826e-dfa1-40f5-c78d-fee9cb862257"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tf.Tensor(3.0, shape=(), dtype=float32)\n",
      "tf.Tensor(6.0, shape=(), dtype=float32)\n"
     ]
    }
   ],
   "source": [
    "x = tf.Variable(1.0)\n",
    "\n",
    "with tf.GradientTape() as t1:\n",
    "    with tf.GradientTape() as t2:\n",
    "        y = x * x * x\n",
    "    dy_dx = t2.gradient(y, x)\n",
    "    print(dy_dx)\n",
    "d2y_d2x = t1.gradient(dy_dx, x)\n",
    "print(d2y_d2x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "qBxk9ly4z60f"
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "colab": {
   "name": "003-automatic_differentiation.ipynb",
   "provenance": [],
   "version": "0.3.2"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
